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ABSTRACT 

Following a review of over 350 evaluation-related 
reports as well as written evaluation activities by the National 
Advisory Council, state advisory coiucils, and over 30 state 
departments of education, three criteria were proposed as methods to 
be used by state education leaders in assessing the worth of an 
existing or proposed evaluation system* These criteria include 
determining: (1) the validity of the system or how accurately the 
data collected by the system reflects the true vocational education 
situation of the state, (2) the effects or impact of tjbe information 
generated by a system on the local program, and (3) the cost of the 
system. .These criteria are discussed along with techniques of 
evaluation, research priorities, and methods of implementing an 
evaluation system. . (SB) 



FILMED FROM BEST AVAILABLE COPY 




ERIC 



us DEPARTMENT OF HcALTH. 
EDUCATION 4 WELFAKE 
NATIONAL INSTITUTE OF 
EDUCATION 

4iS OOCUVENT HAS BEEN REPRQ 
JCtO EXACTLY AS RECEIVED PROM 
•f »*t ©SON OR ORGANIZATION ORIGIN 

iNOlT POINTS Of VIEW OR OPINIONS 
ATLO 00 NOT NECESSARILY REPRE 
NT OPFlClAL NATIONAL INSTITUTE OP 

UCATlON POSITION OR POLICY 



Information Series No, 72 
VT 020 268 



CONDUCTING EVALUATION WITHIN A STATE: 



INFORMATION FOR STATE LEADERS 

D 
D 

D 

D 

J William W. Stevenson 

Assistant State Director and Head 
Division of Research, Planning, and Evaluation 
Oklahoma State Department of Vocational and Technical Education 



William Gary Ward 
Graduate Research Assistant 
Division of Research, Planning, and Evaluation 
Oklahoma State Department of Vocational and Technical Education 



ERIC Clearinghouse on Vocational and Technical Education 
The Center for Vocational and Technical Education 
The Ohio State University 
i960 Kenny Road 
Columbus, Ohio U32IO 

1973 



CONTENTS 



Introduction — • — — 1 

Problem Statement 2 

Micro-Literature Review 2 

Criteria for Systems 5 

Strategies for Implementation 11 

Priorities for Research 12 

Summary 13 

Appendix lU 

Selected Bibliography 17 

Additional Bibliography 22 



MISSION OF THE CENTER 



The Center for Vocational and Technical Education, an independent 
unit on The Ohio Sttte Ifaiversity campus, operates under a grant from 
the National Center for Educational Research and Development, U.S. 
Office of Education. It serves a catalytic role in establishing con- 
sortia to focus on relevaric problems in vocational and technical 
education. The Center is comprehensive in its commitment and respon- 
sibility, raultidisciplinary in its approach and interinstitutional in 
its program. 

The Center^fi mission is to strengthen the capacity of statJ educa- 
tional systems to provide effective occupational education programs 
consistent with individual needs and manpower requirements by: 

Conducting research and development to fill voids in existing 
knowledge and to develop methods for applying knowledge. 

Prograimnatic focus on state leadership development, vocational 
teacher education, curric\ilum, vocational choice and adjustment. 

Stimulating and strengthening the capacity of other agencies 
and institutions to create durable solutions to significant 
problems . 

Providing a national information storage, retrieval and dis- 
semination system for vocational and technical education through 
the affiliated ERIC Clearinghoixse. 



The material in this publication was prepared pursuant to a con- 
tract with the National Institute of Education, U.S. Department of 
Health, Education and Welfare. Contractors undertaking such projects 
under government sponsorship are encouraged to express freely their 
Judgment in professional and technical matters. Points of view or opin- 
ions do not, therefore, necessariOy represent official National Insti- 
tute of Education position or policy. 



FOREWORD 



state leaders faced with operational problems of 
evaluation in vocational and technical education will find 
this publication helpful. 

The authors* review of literature has addressed itself 
to: terminology, use of evaluation, pressures for evalua- 
tion, and conflicts in evaluation. Basic criteria are 
provided for use in selecting an evaluation system. In 
conclusion, the authors indicate directions for future 
research in the area of evaluation. 

The profession is indebted to William W. Stevenson 
and William G. Ward, Division of Research, Planning, and 
Evaluation, Oklahoma State Department of Vocational and 
Technical Education, for their scholarship in the prepara- 
tion of this report. Recognition is also due Jerome Moss, 
Jr., University of Minnesota; and Frank H. Wimer, Washington 
State Coordinating Council for Vocational Education, for 
their critical review of the manuscript prior to final 
revision and publication. Wesley E. Budke, assistant 
director. Information Utilization, The Center, coordi- 
nated the publication's development. 

Robert E. Taylor 
Director 

The Center for Vocational and 
Technical Education 

ERIC Clearinghouse on Vocational 
and Technical Education 
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INTRODUCTION 



After reviewing the written vocational evaluation activities of the 
National Advisory Council, the states * advisory councils, and over 30 state 
departments of vocational education, as well as reading over 350 published 
works on evaluation, it is our judgment that state leaders who are consi- 
dering the implementation or revision of an evaluation system should review: 
(l) the validity of any currently operating or proposed system, (2) the 
effects or impact of the* information generated by a system on the local pro- 
gram, and (3) the cost of the system. An investigation of the validity, 
effects, and cost will aid a state leader in judging the worth of an evalua- 
tion system. 

The three criteria or guidelines are proposed as methods of gaining 
additional decision-m.aking information and should not be considered as the 
only sources of information. 

The needs of the practitioners in vocational education indicate that 
criteria for judging an evaluation system are needed. While there is suffi- 
cient evidence to show that validity, effects, and cost are functional as 
criteria, there is also the hope that this paper will stimulate others to 
thoroughly examine these criteria and generate new and better methods for 
asi:.essing the worth of an evaluation syst^. 

After the problem statement, a micro-review of the literature is pre- 
sented and includes sections on terminology, uses of evaluation result's, pres- 
sures for evaluations, and the potential for conflict in evaluation. A de- 
tailed explanation of validity, effects, and cost follows. Then, strategies 
for the implementation or refurbishing of a system are offered. A section on 
research priorities is presented and a summary completes the written portion 
of the paper. An appendix is included which gives addresses of personnel 
responsible for evaluation in some 30 states. Throughout the paper, current 
evaluation systems are referenced and can be located in the selected biblio- 
graphy. An additional bibliography lists some of the items read but not 
directly referenced. / 
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PROBLEM STATEMENT 



Evaluation procedures and techniques are in a period of transition fron 
first-thought approaches to more mature field-tested systems. Thus state 
leaders are currently faced with a myriad of evaluation systems, and to judge 
which best fulfills the needs -of their state is not a simple task. The pres- 
sures and needs for evaluation and its cohort accountability are present, 
and many are faced with selecting or approving the design of a new system or 
with refurbishing or defending an already existing system. 

These problems require state leaders to make practical decisions on the 
worth of an evaluation system. The careful selection and use of the criteria 
for assessing a system will greatly increase the probability of success. One 
is faced with evaluating an evaluation system, a problem this paper is de- 
signed to address. 



MICRO'LITERATURE REVIEW 

Information theorists refer to error in communications as noise. The 
overall literature on evaluation is extremely noisy, has wide gaps, is diffi- 
cult to follow, is voluminous and contradictory, and can leave a reader in a 
state of frustration. It is easy to understand how phrases such as "raped by 
rhetoric" and "information overloading" could be applied to this literature. 

We have attempted to reduce, to its most communicable form, those ele- 
ments of the documents judged most usable to the practitioner. Our review was 
based on a ccanputer and hand search of the Educational Resources Infozmation 
Center (ERIC), a study of the Review of Educational Research (REP) particular- 
ly the April, 1970 issue, several professional publications, federal laws, re- 
ports of research coordinating units, and state evaluation activities. It is 
highly probable that we missed several important publications. However, based 
upon those that were available to us, the following review is presented* 



Terminology 

The word evaluation appears generally to represent the phenomenon of ex- 
amining data that purports to reflect a situation and judging whether or not 
that situation is acceptable. An evaluation system in its simplest terms re- 
fers to a routinized procedure for the collection, storage, synthesis, packaging 
and communication of evaluation data to the decision-makers who, based on a cri- 
terion, judge the acceptability of the situation. The system is completed when 
the results of the judgpients are returned via a feedback loop for use by other 
decision-makers for causing change in the situation. Validity is used as a 
measure of how accurately the daca collected by the system reflects the true 



situation. The term effects is defined as a measure of the impact of the sys- 
tem on the situation. The cost of the system means somewhat more than Just the 
outlay of resources for its operation; it relates more directly to the pro- 
rating of that cost to the effects of the system. 

We define process evaluation as a measurement of the efficiency of the 
vocational program fron the point of input of resources to the point of output 
of students; product evaluation as a measure of the success of the trained 
student procuring and succeeding on a job congruent with his training; accoun - 
tability as measuring results achieved against the resources expended and fur- 
ther assigning the responsibility for achieving results to specific role in- 
cumbents. We have not been able to attach firm definitions to cost-efficiency, 
cost-effectiveness, and cost-benefit, except to say that each, respectively, 
deals somewhat with a ratio of resources to training activities, products to 
total cost of program, aad the payback to society for its investment in edu- 
cation. 



Uses for Evaluation 

Whil.e many automatically connect evaluation with accountability, there 
appear in reality to be many other uses of evaluation results. One is to ful- 
fill the requirements of state and federal rules and regulations; another is 
for program improvement. Further, there are public relations and competitive 
status for federal funding. There are many uses for evaluation results; how- 
ever, this is not to say that all results should immediately be made public. 
Some results are in-house items and should be used to correct situations prior 
to public disclosure. 



Pressures for Evaluation 

Vocational education is conceptualized as an institution that was created 
by the interaction of o^ir society, culture, government, and already existing 
institutions for the purpose of fulfilling a dynamic need in our civilization-- 
the training of individuals for successful entry into the world of work. Thus, 
from any of those areas, or most likely all of them, could come the demand for 
justifying the expenditures of tax dollars currently being allocated to voca- 
tional education. Further, the call for accountability could originate inter- 
nally from vocational education. Many vocational educators compare the current 
*liigh tide of federal financing" enjoyed by vocational education to the situation 
that encompassed science education after Sputnik in 1957. Their call for ac- 
countability is based on the assumption that if we justify expenditures they will 
not be removed, as was the case in science education. Intermingled with accoun- 
tability is the drive by most professionals involved in vocational education for 
continual improvement and upgrading of all vocational programs. 

Conflict in Evaluation 

The federal and state governments have been attaching evaluation and accoun- 
tability rules and regulations to their allocations, taxpayers have been re- 
jecting local bond issues, and writers in vocational education have been issuing 



stern reccanmendations in favor of evaluation and accountability. On the other 
hand, teachers have been strengthening the barriers of their professional or- 
ganization against the encroachments of those parties on the control of their 
daily work. The following attempts to reflect the potential for conflict. 

Swanson (1970) feels the objectives of education, which reflect cultural 
values, are now econcanic. Lessinger (1970) has observed that "an important 
change has taken place in what Americans expect of their public schools --today, 
the ^questions focus on results obtained for resources used." The "taxpayer 
revolt" and the present difficulty of passage of bonds and levies may show 
evidence of the increased concern placed on results of tax money investments 
by the population. 

The Coleman Report (Schwart'S, 1970) empirically reinforces the economic 
value by indicating that inputs are not accurate measures of schools* contri- 
butions to civilization. Educators, in the past, had been comfortable utilizing 
input data as proof of worth. With the advent of the 70' s, the reflection of 
the economic value by political leaders increased. Peterson (197l)j then Chair- 
man of the Educational Coinnission of the States, vrote, *Ve have measured the 
quality of education on how much money is pouring into it rather than on results. 
It is time to stop and assess where we are and where we are going. " President 
Nixon, in his 1970 Message on Educational Reform, said, *Ve have, as a nation, 
too long avoided thinking of the productivity of schools." The mood may well 
have been expressed when United States Representative Quie said, "I think we are 
going to demand accountability in education from now on in Congress" (Schwartz, 

1970) . Legislators in Pennsylvania, Oklahoma, Colorado, and other states have 
passed or are considering laws on accountability and evaluation. Such a reflec- 
tion of the public view by the political institutions is resulting in a "demand 
for control. " 

In contrast, teachers are seeking increased autonomy and self-direction. 
The increase in advanced degrees, technical competencies, number of male teachers 
support of teachers' organizations, and an intrinsic need to be recognized as a 
full professional were listed by Boyan (1967) as reasons for teacher militancy. 
Tlie 1970 National Education Association Assembly resolutions 70-5 and 8O-I7 
("N.E.A. Resolutions, 1970") and the National Study Conference (Socker and Wilson 

1971) indicate that teachers feel: (l) the expertise of professional educators 
is essential when school programs are evaluated, (2) school evaluation by profit- 
making non-professionals and attempts to transform National Assessment results 
(see Compact , April, 1971) into national standards must be resir >ed, and (3) the 
teacher should be held accountable as a professional to everyone. 

Teachers further expressed the viewpoint that they will only be held fully 
accountable when: (l) the public supports education commensurate with its ex- 
pectations, (2) conditions of teachers* services are jointly defined by boards 
and local teacher associations, (3) the school setting is conducive to teaching 
and learning, and (U) the teaching profession controls its own internal affairs. 
Teachers were urged to discuss the situation with administrators and boards and 
to "adopt a strong resolution on accountability and professional autonomy. " Dar- 
land's observation (Socker and Wilson, 1971) that the teacher is "a most likely 
candidate for scapegoat of the 1970 's," might express the teachers* fears con- 
cerning evaluation and accountability. 



Intertwined with the teachers' drive for autonomy is the heightened 
sensitivity of local schools to increased state and federal involvement. The 
recent federal court's decisions on the lack of parity education that results 
from the curreiit property tax system of financing schools was viewed by some 
as a final blow to local control. The emphasis on evaluative activities could 
be considered as a move toward additional state and federal control. 

Assuming this is true, a state leader could be caught between the demand 
for control on one side and the drive for autonomy on the other. However, this 
is not a new situation for state leaders. Further, these reviewers raise the 
question: Just how widespread are the conditions for conflict? It would seem 
prudent for a state leader to investigate the potential for conflict between 
teachers and their professional organizations, school districts, and state de- 
partments. The situation may or may not exist. 

An example may clarify this proposition. Under the 1968 Amendments (U.S. 
Congress, 1968) to the 1963 Vocational Education Act, each state and trust 
territory must establish an advisory council. One of the primary duties of 
this council is to yearly evaluate the programs, services, and activities of 
the vocational offerings of its state and to report those ^tndings to the 
National Advisory Council by October of each year. The National Advisory Coun- 
cil in turri generates a report to the U.S. Commissioner of Education. That 
procedure is the federally mandated evaluation of vocational education. However, 
in the National Advisory Council's Fifth Annual Report (1971) ♦ tney wrote in 
essence, "is anyone listening to our reports?'' This is an example"* of a required 
evaluation system, which apparently is not used in the dec is ion -making process 
by those who required it. It will be of interest to see the effects of that 
system on new legislation. 

Another proposition arises when one considers the vocational teacher. There 
is some evidence to indicate that reimbursed teachers are quite skillful in 
"playing off" a state department representative against a local school adminis- 
trator in order to achieve the resources and policies they desire for their pro- 
gram. A seasoned vocational teacher might be very enthusiastic about the visi- 
tation of an evaluation team. 

These propositions were not made for the sake of contradicting the informa- 
tion presented und'jr the Conflicts in Evaluation section but rather to illustrate 
the difference in v/hat organizations say and what people do. 



CRTTERIA FOR SYSTEMS 

There are at least three characteristics of a system which could be measured 
to determine the worth of that system: (l) validity, (2) effects, and (3) cost. 
The level or score of those measures alone will not tell a state leader whether 
he should accept or reject a system. The situation in each state and the pur- 
poses for which the evaluation results will be used must be considered when one 
establishes an acceptance level for each characteristic. 



validity 



The first question we shoiad ask in appraising a system is '"Does the 
system collect the types of information we ^an use in realistically under- 
standing the situation?" The most appropriate guidelines for answering that 
question are prior knowledge and common sense. There arc four broad areas 
a comprehensive system could investigate: (1) the vocational training needs 
of the state, (2) the efficiency of the evaluation process, (3) the success 
of the product, and {k) the congruency of the product to the needs of a state. 
Inside each area a state leader could specify the exact information he de- 
sires, such as, the number of minority group members currently enrolled, grad- 
uated, or placed on Jobs. In most of the systems reviewed for this paper, 
these types of questions were normally generated by a committee, thus estab- 
lishing a type of face or construct validity. 

After those decisions are made, one ^,hen becomes concerned with the so\ind- 
ness of the methodology used in gathering the data and the accuracy of the 
measuring instruments used to collect tlvat data. The way the information is 
collected, called design, and the accuracy of the measuring instruments, called 
validity and reliability, could probably best be investigated by a researcher 
experienced in coveting data on a large scale. However, it would seem wise 
for the state leader to have seme knowledge about the des5.gn and instrumentation 
currently being used and some of the problems one might encounter. 

Vocational Training Needs in a State . In measuring this first broad area 
of concern, the Occupational Training Information System (OTIS) - (Oklahoma 
State Department, 1972) can be used to measure specific Job market needs. In 
cooperation with a state's employment security agency, it collects current and 
predicted Job demand information on designated occupations from business and 
industry. This is accomplished via interviews and mail-out questionnaires to 
a representative sample of employers. On the other side, the supply of skilled 
trainees and their availability rate Is collected trom public and private training 
programs. The supply and demand are then interfaced, thus generating a numerical 
list of areas where training is needed. OTIS measures objective visable items, 
such as jobs which can be directly measured. Suppose under the vocational 
training needs of a state, one desired information of a more subjective nature, 
thus more difficult to measure, for example, '"What are the current values of the 
disadvantaged students toward work?" That would involve a study o£ work ethic, 
religion, and subcultural values, et cetwa, which would require special instru- 
mentation, field tests, and pilot studies. Generally one can place greater con- 
fidence on the results of an objective study than on the results of a subjective 
study. 

Efficiency of the Evaluation Process . Process evaluation was found to be 
the most common component of the systems now used by the states. There are at 
least four broad procedures currently being used to identify those occurrences 
that cause quality (Stevenson, 1970): (1) school visitations by Judges:, (2) 
self-evaluation by the local school, (3) utilization by the state department of 
already reported data, and (h) cost analysis. The validity of many of those 
procedures is questionable, yet there is insufficient evidence to say such methods 
are wrong. 
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In school visitations by judges, the most conanon measuring devices u^ed 
are rating scales or category instruments. They are used to record the judge's 
observations on equipment, facilities, educational practices; and at times ^ 
teacher behavior, cost, administration, curriculum, and philosophy. Rating 
scales are classified as high- inference (highly subjective) measures by Rosen- 
shine (1970) and usually require responses on a 1-5 or 1-7 point Likert-type 
scale. The disadvantages of rating scales are: the halo effect, error of 
central tendency, leniency error, lack of a ccranon referent for scoring cali- 
brations such as "excellent" or "seldom" (Mouly, 1969), and the difficulty of 
translating the results of the instruments into .specif ic behavior. However, 
rating scales do offer a great deal of flexibility, and when validity and inter- 
judge reliability (consistent agreement on the meaning of re-ponses) are es- 
tablished, they can be quite useful as data gathering instruments. Few of the 
reviewed rating instruments showed any evidence of the establishment of validity 
or reliability. 

category systems are classified as low-inference (more objective) measures 
by Gage (1969) because the items focus on specific, denotable, relatively ob- 
jective responses which are recorded by the observers as frequency count's under 
predetermined blocks of interest. Major disadvantages include the fact that 
judges require ti'ining to use the instrument, the observations on the more sub- 
jective portions muy have to be reported several times, and while several cate- 
gory instruments already exist for observing teachers, none could be located 
that will measure the overall program. 

Two problems arise in interpreting results from rating scales and category 
instruments. The data are usually of the nominal or low ordinal level, thus 
limiting the available tools of statistical analysis (Kerlinger, 1964). Further, 
to what criterion will the resuJ.ts be compared? What is an acceptable level? 

While the current level of measurement in the classroom is inexact, there 
are other benefits to the visitation of judges. One is that the teacher or 
administrator who knows the judges are coming may correct, prior to the visit, 
items he knows are wrong. The other is that if the judges are university 
teachers, state department personnel, advisory council members, and the like, 
they might, due to the visit, become more attxmed to the problems of the local 
schools. 

An alternative to the school visitation by judges is the completion of a 
measuring device by the state supervisor or coordinator, usually in cooperation 
with tlie local instructor. Alaska and Virginia have used this technique. 

Self -evaluation by the local school and the reporting of those results to 
the state department is another way to gain information about the process of 
education. This procedure is typified by Byraxn's (1971) approach of a locally- 
directed evaluation and the Kentucky project (university of Kentucky, 1970). 
The states of Arkansas, Connecticut, Iowa, Mississippi, Nevada, Tennessee, and 
Vermont have used similar procedures. An alternative method is the requiring 
of local programs to report empirical data on students and programs. This al- 
ternative is exemplified by Starr (1970) and Frazier (1971). The validity of 
these techniques are in direct proportion to the accuracy of the measuring de- 
vice and the attention to detail of the local school. 
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The analysis of data already being reported by the local schools is a 
third method for gaining process information. While this would probably not 
reveal a great deal of information, it could be used to supplement a system. 
These types of information should definitely be reviewed prior to the opera- 
tionalization of a system in order to avoid collecting the same information 
twice. Here again, the validity is in direct proportion to the accuracy of 
the measuring device and the attention to detail of the local school. The 
fourth and newest technique for gaining information on the process is cost 
analysis. Cost studies normally associate the cost of the process to the 
product of the program, and depending on the technique, may be called cost 
efficiency or cost effectiveness (Kauftaan, 1968). Chumbley and Hopkins (1972) 
have accomplished state-wide average cost of selected programs. This aUows 
one to compare the cost of the process of one program to a state-wide average 
of the same type of program, and while generating valuable planning informa- 
tion, also opens the door for establishment of a Program, planning, and Bud- 
geting System. The validity of cost analysis is in the process of beccxning 
well established and has the potential for developing into an objective system 
of evaluation. However, it appears that additional types of information on 
the process may be necessary for a fuller understanding of the siti^tion. 
According to Sjogren (1970), the works of Warmbrod (1968), Kotz (19o7a, ISoTb), 
and Thomas (1969) are excellent sources for learning about the relationship 
of economic concepts to educational evaluation. 

The efficiency of state departments of education and other service sup- 
port units are, in effect, being evaluated when the local schools are eval- 
uated. Yet, it is difficult to "break-out" (Barro, 1970) the exact contri- 
butions of those agencies. If the state department has not yet adopted a Pro- 
gram, planning. Budgeting and Evaluation System, or a Management by Objectives 
System, then Swanson (1967a-b), Lee (I967), Oregon. .. (1968), and Rice (1968) 
could be utilized as foundations for internal evaluation. 

Product Success . We move now from the validity of process evaluation to 
a study of the third broad area of product evaluation. Hard empirical measures 
of the outputs of vocational education have been developed in the form of 
student follow-up (fTazier, 1971; Cox, 1969; or Schultz, 1969), joh related- 
ness (Wheeler, 1971), cost analysis (Hopkins, 1970), and OTIS (Oklahoma State 
Department, 1972). The states of Arizona, Oklahoma, Colorado, Florida, Iowa, 
Michigan, Minnesota, Mississippi, Missouri, Nebraska, New York, Massachusetts, 
Ohio, South Carolina, South Dakota, Tennessee, Wisconsin, and many others cur- 
rently follow up students for the purpose of obtaining data on output. The 
technique of moving past job placement and on to how well the training re- 
ceived is related to the obtained job is typified by Wheeler (1971) and varia- 
tions have been used in Florida, Nebraska, and Ohio. There is also the possi- 
bility of using standardized tests (Davis, 1968), labor union evaluations, and 
licensing board tests as measures of efficiency/effectiveness. The technique 
one selects depends upon what one considers as a measure of quality (Gtevenson, 
1970). 

The validity of most student follow-ups are not as high as one might wish. 
There are generally two ways for collecting data; both require the maintenance 
of records on permanent addresses or homes for each student. One is to require 
the teacher to contact the graduates on, for example, a one, three, and five- 
year follow-up. The second is direct mailing from a state department to the 
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student. Validi.ty checks have indicated that while a few teachers tend to 
manipulate the results, the graduates, when responding on questionnaires, 
tend to inflate the aspects of their current jobs. Students also have a 
lower return rate on direct mail questionnaires. While neither follow-up 
system is perfect, they are the best that have yet been developed. One 
should inspect the questionnaire used to determine if it gathers the types 
of information desired, such as, job relatedness, mobility, salary, prc»no- 
tion, and recommendations for program change. Further, it would be advisable 
to conduct a "bias check" on the non-respondents. Usually a few graduates 
who did not respond can be located by phone. It is indeed interesting to 
learn why some do not respond to questionnaires. 

If cost data were collected during the proces evaluation, it is now 
possible to compute a cost per graduate placed. It ppears that soon, based 
on current research and development programs, one may even be able to gener- 
ate, with some resdism, a cost-benefit analysis based on the graduates 
placed and resource outlay. 

Needs of the State and the Product . The measuring of the congruency of 
the outcomes or products of the vocational programs to the needs of the 
state is the last area to be discussed xinder validity. A culmination of all 
the measures are now compared — needs against output With the exception of 
attempting to compare unlike items, or inferring too much from subjective or 
poor measures, little can go wrong in connection with validity. The system 
has arrived at a point where the state leaders, based on the criteria of 
success that they established for the vocational program, can judge how well 
those criteria were met. Stake (1970) points out that "success does not 
mean hitting the bulls-eye; success means coming acceptably close to a val- 
ued target . " 

We have invested several pages in validity because without validity 
there is little need for concern with effects and cost. Yet, validity alone 
does not make a realistic system. The effects and cost are vitally impor- 
tant criteria. 



Effects 

The concept of effects can be tailored to assist in measuring the worth 
of any system. Essentially it is a method of measuring the impact of the 
system on the situation in the local program. For example, if the purpose 
of the evaluation system is to cause program improvement , then one could mea- 
sure the changes in the situation which were caused or influenced by the 
results of the evaluation system. If the purpose was public relations, one 
could measure how many more people in £ specific population have adopted a 
more favorable attitude toward vocational education. Generally, whatever the 
purpose, an e^/^aluator or researcher can devise an instrument ths.^*; w: tl test 
how well the system fulfilled its purpose. 

There are at least three other factors that, while extraneous to the 
data collection aspects of the system, can threaten a realistic appraisal of 
the effects. The first deals with the packaging of evaluation results, an- 
other with the method of delivering those results, and the third is concerned 



ERLC 



9 



(depending on the system) with assisting the local teachers and adminis- 
tration in implementing the recommendations of the evaluation. On the 
packaging of evaluation results, the literature is poorly developed and only 
suggests that the communication be kept short and simple. 

The procedure of delivery or "feeding back" all pertinent data for de- 
cision-making is typified by the Massachusetts Information Feedback System 
for Vocational Education (Conroy, 19^9) and Rzonca and Tcxnlinson's (1971) 
Educational Data System. These procedures logically prescribe methods for 
collecting, storing, and returning data, as desired, to the decision-maker. 
Copa (1971) establishes guidelines for decision-making when a data bank is 
used and Coster and Morgan (1969) discusses the role of evaluation in the 
decision-making process. 

When assisting the local teachers and administration in implementing 
the recommendations, one ventures into the area of supervision, which is, by 
its nature, an interpersonal situation laced with tradition, policy, rules, 
and usually problems. However, to steer away or to be timid in helping the 
local program introduce change is to have the entire system fail because of 
a lack of follow-through. 



Cost 

The computation of the cost for a system has been simplified and about 
the only problem is to decide the exact point at which to stop charging-off 
cost. Answers are needed for questions such as, should a portion of the e- 
valuator's bosses' salaries be charged off as supeirvision of the system, or 
if the judges used in school visitations are state depajrtment personnel, 
should a portion of their salaries be charged-off to these systems. It would 
appear that decisions of this nature would depend on the situation, and in 
reporting the cost^ it would be more communicable if the state identified ex- 
actly where they stopped charging-off cost. That would allow states to com- 
ppxe cost by simply adjusting the cost to where both were measuring the same 
items. 

There appear to be seven areas of cost: Salary and benefits of the e- 
valuator and his professional staff per year; travel of staff and judges; 
salary and benefits of supportive personnel per year or a proportion based 
on contact time (it should include secretaries, ccanputer service personnel, 
consultants, and might include scxne supervision and judges' salary charge- 
off); depreciation cost of offices or amount of rent (depreciation on govern- 
ment buildings could be canputed by a straight-line depreciation); deprecia- 
tion cost and rental cost of equipment would include items such as canputer 
time and a depreciated per year cost of equipment such as typewriters , cal- 
culators, and furnishings; operation and maintenance of offices would include 
cost of office supplies, utilities, janitor supplies and maintenance of the 
office, printing cost, and phone; insurance on office and content. 

After those are totaled and a one year's cost is computed, the cost to 
the effects measure co^iLd then be prorated; such as, if the measxire was 
amount of change, each change cost could be determined. For those systems 
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that process all of their information through a computer, a cost per record 
could be calculated. There are several comparisons to be generated with 
the cost data. 



STRATEGIES FOR IMPLEMENTATION 



Since each state faces unique political and financial situations, we 
can do little in this section except interface some current management tech- 
niques with some successful methods of implementation and draw from that 
marriage a few guidelines for implementation. Each state leader will be 
able to enrich the following guidelines from his own experience. 

Whether the state desires to begin an entirely new evaluation system or 
refurbish an already existing system, one is involved with change. The 
change process normally requires formative leadership as opposed to the some- 
what more routine manner of administration or management (Carver and Sergio- 
vanni, I969; Halpin, 1958 )• The guideline here would* appear to be that those 
persons who hold the uppermost job positions in the state's educational or- 
gsmizations with the backing of their power structure must openly commit 
themselves to the acquisition of a valid, effective, and economical evaluation 
system. However, prior to the open commitment, it would be wise for the 
state leader to informally canvas the opinion leaders (Bice, 1970; Teague, 
1971) of those groups (teachers, administrators, professional organizations) 
who will have a vested interest in evaluation. It might allow for a measure 
of the opposition, and if it is massive, one could take appropriate steps. 
Generally the gaining of those opinion-leaders' willingness to cooperate can 
be accomplished by offering inducements, based on what they value, to gain 
their contribution of cooperation (Barnard, 1933). Once the commitment an.d 
support for evaluation has been developed, the decisions as to the lanner of 
evaluation and its finarj^cing must be made. 

Group decision-making, or at least group input to decision-making, appears 
to be the current method in education for arriving at decisions that effect a 
very wide spectrum of education in a state. This collegial decision-making 
has inherent within itself a way to insure that the individuals within the 
states' organizations will support the rules and regulations of those organi- 
zations. The making of rules by individuals, hopefully opinion-makers from 
all levels, tends to cause the groups they represent to enforce and abide by 
the organizational rules. In this case, it means that if teachers, adminis- 
trators, board and advisory council members, via the democratic process, 
arrive at the methods — or how to evaluate based on the three criteria of vali- 
dity, effects, and cost — they will be more inclined to support -che evaluation 
system developed by those groups they represent. This further means that the 
results of the system will be more closely regarded as a guide for change in 
the local programs. 

The administrator attempting to install an evaluation system should 
szTOzs the positive aspects of effective evaluation. The system can contri- 
bute to the improvement of programs from the aspect of all individuals in- 
volved. The evaluation system can support the teacher in his need for 
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improved facilities and equipment, stronger administrative support, better 
student selection procedures, and the like. The system can assist the 
local administrator in changes and improvements he wishes to make in pro- 
grams. The evaluation report can be a useful tool to supervisors in work- 
ing with teachers. Evaluation can help administrators demonstrate to the 
public and the legislature that honest efforts at accountability are taking 
place. If these benefits are stressed by those promoting evalu^ition, ac- 
ceptance can be increased. 

There are provisions in the state plan for evaluation costs. Many 
state advisory councils share their evaluation money with state depart- 
ments, and in some cases (New York and Massachusetts) states have combined 
to operate evaluation programs. Other techniques for implementation may 
be located in Reynolds (I967), Divita (I968), and Swanson (I966). 



PRIORITIES FOR RESEARCH 

If the resources invested in evaluation are to have a long-range ef- 
fect, then it is necessary to identify those activities in the process of 
education that cause quality. On the surface, this appears to be a type of 
research that begins with measuring the correlation between activities in 
the process and those measures of success in the product. The measures of 
success of the product are usually those items, such as placement or job 
relatedness, that are judged by state leaders as measures of quality. An 
example may clarify the question: Does the existence of local advisory 
councils correlate with a high placement rate? After several correlation 
studies have been replicated in several question areas, then we could begin 
explicit cause and effect studies; such as, **Does the existence of local 
advisory councils cause a high placement rate?" 

The area of communication between practicing evaluators in each state 
is at an extremely poor state of development. A way for these practitioners 
to share their experience should immediately be developed. 

Under the heading of Effects , we recognized the problems of packaging, 
delivery, and follow-through on evaluation results. These areas probably 
should be researched inside each state. The number of intervening variables 
and different situations faced by each would tend to preclude generalizing 
from one state to a dissimilar one. 

An additional area of investigation is the location of the job position 
of the evaluator in the hierarchical organization of the state departxiaent . 
An evaluator would seem to require a certain degree of professional freedom 
in the measurement of educational activities. With very little evidence, it 
is felt that the degree of professional freedom is in proportion to the dis- 
tance of the job position from the center of the organization. The closer 
to the center of the organization, the more the evaluator will be under the 
control of his superordinates. However, as the distance from the center of 
the organization increases, his authority decreases. What is the optimal 
location? Does it vary from state to state? 
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SUMMARY 



This paper has specified three ways to evaluate an evaluation system. 
The first is the validity of the system which was defined as a measure of 
how accurately the data collected by the system reflects the true voca- 
tional education situation of the state. The second, called effects ^ was 
defined as a measure of the impact of results of evaluation on the voca- 
tional programs in a state. Third was the prorating of the cost of the 
system to the intended effect of the system or to a record- input cost. 



It was further noted that the potential for conflict between the pres- 
sure for evaluation by elected officials and the drive for autonomy by 
teachers and local schools could place a state department of education in a 
position to draw fire from both sides. 

Several current techniques of evaluation were discussed along with re- 
search priorities and methods of implementation. It appears that, on a 
whole, evaluation systems are in a state of transition from research devel- 
opment to full realistic use. Discretion must be used and flexibility must 
be maintained when a state leader operationalizes an evaluation system. 
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APPENDIX 



Contacts by State 



Alaska : 



Arizona: 



Arkansas : 



California: 



Colorado : 



Connecticut: 



Delaware: 
Florida: 
Illinois : 



Iowa: 



Massachusetts 
and New York: 



Director of Vocational and Adult Education, Department of 
Education, Division of Vocational Education, Pouch F - 
Alaska Office Building, Juneau, Alaska 998OI 

Assistant Director for Supervision and Evaluation, State 
Department of Vocational Education, 1333 West Camelback 
Road, Phoenix, Arizona 85OI3 

Administrators of Program Planning, Coordinators, Evalua- 
tion, and Exemplary Programs, Department of Education, 
Division of Vocational, Technical, and Adult Education, 
Arch Ford Education Building, Little Rock, Arkansas 
72201 

Research and Evaluation Consultant, Department of Educa- 
tion, Vocational Education Section, Research Coordinating 
Unit, 721 Capitol Mall, Sacramento, California 95B1U 

Supervisor of Evaluation, State Board for Community Col- 
leges and Occupational Education, 207 State Services 
Building, 1525 Sherman Street, Denver, Colorado 80208 

Research Coordination Unit, Bureau of Vocational Services 
Division of Vocational Education, Connecticut State De- 
partment of Education, P.O* Box 2219, Hartford, Connecti- 
cut 06115 

Supervisor, Occupational Research Section, State Depart- 
ment of Public Instruction, Dover, Delaware 19901 

Department of Education, Vocational Research and Evalua- 
tion Function, Tallahassee, Florida 32301+ 

'^hree Phase System for State-wide Evaluation of Occupa- 
tional Education Programs." Bulletin No. 35-971. Pro- 
gram Approval and Evaluation Unit, 1035 Outer Park Drive, 
Springfield, Illinois 62706 

Director, Career Education Division, Department of Public 
Instruction, Grimes State Office Building, Des Moines, 
Iowa 50319 

Evaluation Service Center for Occupational Education. Re 
search Coordinating Unit Director, The University of the 
State of New York, The State Education Department, Bureau 
of Occupational Educational Research. Albany, New York 
I222U 
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Michigan: 



Research Consultant, Department of Educatlcn, Bureau of 
Research, Lansing, Michigan U8902 



Minnesota: Division of Vocational and Technical Education, Centen- 

nial Office Building, St. Paul, Minnesota 55101 

Mississippi: Assistant Director, Program Planning and Evaluation, De- 

partment of Education, Divis.ion of Vocational and Techni- 
cal Education, P.O. Box 771, Jackson, Mississippi 39205 

Missouri: Assistant Commissioner, Director, Vocational Education, 

Department of Vocational Education, Jefferson City, Mis- 
souri 65101 

Nebraska: Executive Director, Nebraska State Advisory Council for 

Vocational Education, 307 AH East Campus, University of 
Nebraska, Lincoln, Nebraska 68503 

Nevada: Department of Education, Vocational-Technical Education 

Branch, Carson City, Nevada 39701 

New Hampshire: Director of Research Coordinating Unit, Department of 

Education, Division of Vocational-Technical Education, 
Stickney Avenue, Concord, New Hampshire 03301 

North Dakota: State Office Building, Vocational Education, 9OO East 

Boulevard Avenue, Bismarck, North Dakota 5S5OI 

Ohio: The Center for Vocational and Technical Education, Divi- 

sion of Evaluation, The Ohio State University, I96O Kenny 
Road, Colimibus, Ohio kS210 

Oklahona: Evaluation Coordinator, Oklahoma State Department of Vo- 

cational and Technical Education, 1515 West Sixth Avenue, 
Stillwater, Oklahoma 7^07^ 

Pennsylvania: Bureau of Curriculum Development and Evaluation, Division 

of Evaluation, Department of Public Instruction, Box 9II, 
Harrisburg, Pennsylvania 17126 

South Carolina: Assistant Director, Office of Vocational Education, State 

Department of Education, Coliimbia, South Carolina 29200 

South Dakota: Division of Vocational and Technical Education, State 

Capitol, Pierre, South Dakota 57501 

Tennessee: Coordinators of Program Planning and Evaluation, Depart- 

ment of Education, Division of Vocational-Technical Educa- 
tion, 205 Cordell Hull Building, Nashville, Tennessee 
37219 
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Utah: 



Vermont : 



Virginia : 
West Virginia: 



Planning Unit, Office of the Utah State Superintendent of 
Public Instruction, lUOO University Club Building, Salt 
Lake City, Utah 8UIII 

Assistant State Director of Vocational Education, Depart- 
ment of Education, Vocational and Technical Division, 
State Office Building, Montpelier, Vermont O5602 

Director of Vocational Education, State Board of Educa- 
tion, Richmond, Virginia 23216 

Program Specialist, Program Development Unit, Department 
of Education, West Virginia State Board of Education, 
Charleston, West Virginia 25305 



Wisconsin: Research Supervisor, Board of Vocational, Technical and 

Adult Education, 137 East Wilson Street, mdison, Wiscon- 
sin 53703 
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